FULL FINE-TUNING

MoExtend: Tuning New Experts for Modality and Task Extension (2024.acl-srw)

Challenge:	AdapterHub framework enables dynamic “stiching-in” of pre-trained adapters for different tasks and languages.
Approach:	They propose a framework that allows dynamic "stiching-in" of pre-trained adapters for different tasks and languages.
Outcome:	The proposed framework allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages.

Copied to clipboard

Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Zitnik, Pan Zhou

Multimodality

Challenge:	Existing instruction tuning methods for large language models (LLMs) are costly and difficult to implement.
Approach:	They propose a framework to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models.
Outcome:	The proposed framework enables rapid adaptation and extension to new modal data or tasks without tuning pretrained models.

Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding (2026.acl-srw)

Copied to clipboard

Ayaan Siddiqui

Multimodality

Challenge:	Existing models for understanding figurative language in images perform well on literal recognition but fail on multimodal figurativ benchmarks.
Approach:	They propose a model that adapts to idiomatic and figurative language using literal alignment bias rather than limited model capacity.
Outcome:	The proposed model generalizes across five idiom-rich languages despite being trained on English supervision.

Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning (2023.emnlp-demo)

Copied to clipboard

Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer

Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients (2025.acl-long)

Challenge:	Adapters is an open-source library that unifies parameter-efficient and modular transfer learning in large language models.
Approach:	They propose to integrate 10 different methods into a unified interface for parameter-efficient and modular transfer learning in large language models.
Outcome:	The proposed library is able to perform on multiple NLP tasks and is open-source.

Copied to clipboard

Jabin Koo, Minwoo Jang, Jungseul Ok

Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages (2025.acl-srw)

Challenge:	Existing methods for federated fine-tuning for Large Language Models suffer from performance degradation at low ranks in heterogeneous data settings.
Approach:	They propose a low-rank adaptive model with Alternating freeze and Adaptive rank selection which reduces the number of uploaded parameters by 99.8% .
Outcome:	The proposed low-rank Adaptation maintains robustness even under extreme heterogeneity and low rank conditions while preserving communication efficiency.

Copied to clipboard

Daniil Gurgurov, Ivan Vykopal, Josef Van Genabith, Simon Ostermann

AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2021.eacl-main)

Challenge:	Low-resource languages (LRLs) face significant challenges in natural language processing due to limited data.
Approach:	They evaluate adapter-based methods for adapting mLMs to low-resource languages . they use unstructured text and structured knowledge from ConceptNet to evaluate adapters .
Outcome:	The proposed methods outperform large language models and LLaMA-3 and deepSeek-R1 models on low training data.

Copied to clipboard

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych

Pass-Tuning: Towards Structure-Aware Parameter-Efficient Tuning for Code Representation Learning (2023.findings-emnlp)

Challenge:	Existing methods for incorporating knowledge from multiple tasks suffer from catastrophic forgetting and difficulties in dataset balancing.
Approach:	They propose an algorithm that extracts and combine adapters in a knowledge composition step.
Outcome:	The proposed class outperforms traditional methods such as full fine-tuning and multi-task learning on 16 diverse NLU tasks.

Copied to clipboard

Nuo Chen, Qiushi Sun, Jianing Wang, Xiang Li, Ming Gao

Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models (2025.findings-acl)

Challenge:	Code pre-trained models have been proposed and widely applied in the domain of code intelligence.
Approach:	They propose a method that uses a plug-and-play graph neural network module as a tunable prefix to exploit structural information of source code.
Outcome:	The proposed method exploits structural information of source code and could replace full fine-tuning.

Copied to clipboard

Di Wu, Xin Lu, Yanyan Zhao, Bing Qin

Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)

Challenge:	Large language models achieve effective safety alignment at the time of release, but fine-tuning often compromises safety mechanisms.
Approach:	They propose a method that performs safety realignment for large language models . they identify unsafe delta parameters from the fine-tuned models and recalibrate the retained parameters .
Outcome:	The proposed method improves safety performance on safety benchmarks and jailbreak attacks while maintaining their performance on downstream tasks.

Copied to clipboard

Chenxi Whitehouse, Fantine Huot, Jasmijn Bastings, Mostafa Dehghani, Chu-Cheng Lin, Mirella Lapata

Information Extraction & Text Mining

Challenge:	Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning.
Approach:	They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored.
Outcome:	The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks.

SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models (2024.findings-naacl)

Copied to clipboard

Hossein Hajipour, Ning Yu, Cristian-Alexandru Staicu, Mario Fritz

Sort by Structure: Language Model Ranking as Dependency Probing (2022.naacl-main)

Challenge:	Large datasets are increasingly available for pre-training source code models, but obtaining representative training data that fully covers the code distribution for specific downstream tasks remains challenging due to the task-specific nature and limited labeling resources.
Approach:	They propose a systematic approach that simulates various OOD scenarios along different dimensions of source code data properties and investigates model behavior under different fine-tuning methodologies.
Outcome:	The proposed approach simulates various OOD scenarios along different dimensions of source code data properties and exposes multiple failure modes attributed to OOD generalization issues.

Copied to clipboard

Max Müller-Eberstein, Rob van der Goot, Barbara Plank

SparseFit: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations (2024.acl-long)

Challenge:	Existing algorithms for pre-trained language models lack performance indicators for linguistic tasks such as structured prediction.
Approach:	They propose to measure the degree to which labeled trees are recoverable from an LM’s contextualized embeddings by probing to rank LMs for parsing dependencies in a given language.
Outcome:	The proposed approach predicts the best LM choice 79% of the time using less compute than training a full parser.

Copied to clipboard

Jesus Solano, Mardhiyah Sanni, Oana-Maria Camburu, Pasquale Minervini

SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters (2022.findings-emnlp)

Challenge:	Models that generate natural language explanations (NLEs) for their predictions often require large datasets of human-written NLEs at training time, which can be expensive and time-consuming to collect.
Approach:	They propose a sparse few-shot fine-tuning strategy that leverages discrete prompts to jointly generate predictions and NLEs.
Outcome:	The proposed approach compares sparse few-shot fine-tuning with existing parametric fine- tuning techniques on three sizes of the T5 language model and four datasets and produces competitive results for both task performance and NLE quality.

Copied to clipboard

Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, Dacheng Tao

Simple, Scalable Adaptation for Neural Machine Translation (D19-1)

Challenge:	Pretrain-finetuned models are increasingly complex and require more parameters to match the performance of full fine-tuning.
Approach:	They propose an efficient Adapter Tuning technique that freezes pretrained language models and fine-tunes a few extra modules.
Outcome:	The proposed setting outperforms the standard Adapter Tuning by 80% . the proposed setting is easy to use and has a high sparse ratio .

Copied to clipboard

Ankur Bapna, Orhan Firat

A Weak Supervision Approach for Few-Shot Aspect Based Sentiment Analysis (2024.eacl-long)

Challenge:	Recent advances in deep learning have led to significantly improved quality on Neural Machine Translation (NMT) however, performance on out-of-domain data or low resource languages remains poor.
Approach:	They propose a simple yet efficient approach for adapting pre-trained models to multiple tasks simultaneously.
Outcome:	The proposed approach is on par with full fine-tuning on domain adaptation and massively multilingual NMT on a massively multilingual dataset.

Copied to clipboard

Robert Vacareanu, Siddharth Varia, Kishaloy Halder, Shuai Wang, Giovanni Paolini, Neha Anna John, Miguel Ballesteros, Smaranda Muresan

Sentiment Analysis

Challenge:	Existing methods to improve few-shot performance in aspect-based sentiment analysis (ABSA) require complex interactions between the target and the polarity of the sentiment.
Approach:	They propose a pipeline approach to construct a noisy ABSA dataset and adapt it to the ABSA tasks.
Outcome:	The proposed model outperforms the state-of-the-art on the aspect extraction sentiment classification task and is capable of performing the harder aspect sentiment triplet extraction task.

Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)

Copied to clipboard

Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun

Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation (2026.acl-long)

Challenge:	Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models.
Approach:	They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities.
Outcome:	The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning.

Copied to clipboard

Dianyun Wang, Qingsen Ma, Yuhu Shang, Zhifeng Lu, Zhenbo Xu, Lechen Ning, Huijia Wu, Zhaofeng He

Explainability & Interpretability in NLP

Challenge:	Prior work has shown that safety behaviors are governed by low-rank structures . Low-Rank Adaptation (LoRA) consistently underperforms full fine-tuning and reinforcement learning on safety benchmarks .
Approach:	They propose a safety alignment system that disentangles safety-relevant directions into monosemantic features and constructs an interpretable safety subspace from SAE directions.
Outcome:	Empirically, the proposed model achieves 99.6% safety rates across multiple model families and scales . low-rank Adaptation consistently underperforms full fine-tuning and reinforcement learning on safety benchmarks compared with previous methods .

Parameter-Efficient Fine-Tuning without Introducing New Latency (2023.acl-long)

Copied to clipboard

Baohao Liao, Yan Meng, Christof Monz

AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models (2022.findings-emnlp)

Challenge:	Parameter-efficient fine-tuning of pre-trained language models has been demonstrated to be effective, but its inherent characteristics limit its performance.
Approach:	They propose to generate a sparse mask in a task-agnostic manner by modifying only a small subset of existing parameters and adding new parameters.
Outcome:	The proposed method surpasses existing methods on the GLUE benchmark by a significant margin.

Copied to clipboard

Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, Dongsoo Lee

SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (2025.findings-acl)

Challenge:	Existing approaches to improve inference efficiency by accelerating model fine-tuning have not been thoroughly explored.
Approach:	They propose to combine parameter-efficient adaptation and model compression to accelerate model . they propose to freeze binary parameters and scale scaling factors for target tasks .
Outcome:	The proposed algorithm achieves >10x compression ratio under 4-bit quantization and >1,000x reduction in trainable parameters.

Copied to clipboard

Chenhao Ding, Jiangyang Li, SongLin Dong, Xinyuan Gao, Yuhang He, Yihong Gong

MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning large language models (LLMs) introduce parameter interference, leading to a gap in generalization performance for specific tasks compared to full fine-uning.
Approach:	They propose a parameter-separated low-rank adapter to account for task differences by decomposing LoRA’s parameter matrix into multiple independent subspaces and assigning them differentially to distinct tasks.
Outcome:	The proposed method outperforms LoRA in trainable parameter efficiency and overall model performance on various NLP tasks.

Copied to clipboard

Ting Liu, Zunnan Xu, Yue Hu, Liangtao Shi, Zhiqiang Wang, Quanjun Yin

Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models (2024.acl-long)

Challenge:	Existing methods for Referring Expression Comprehension (REC) lack specific domain abilities for precise local visual perception and visual-language alignment.
Approach:	They propose a framework for Parameter-Efficient Transfer Learning to localize a visual region via natural language using a prior-guided prior.
Outcome:	The proposed framework achieves the best accuracy compared to the current methods with only 1.41% tunable backbone parameters.

Copied to clipboard

Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang

GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs (2025.acl-long)

Challenge:	Existing methods of model editing and knowledge updating add additional network parameters, knowledge bases, knowledge base, and model parameters.
Approach:	They propose a new paradigm for fine-tuning called F-Learning that employs parametric arithmetic to facilitate the forgetting of old knowledge and learning of new knowledge.
Outcome:	The proposed model outperforms existing models on two datasets and is comparable to full fine-tuning and LoRA fine-uning.

Copied to clipboard

Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Maria Krylova, Venediktov Egor, Zuev Aleksandr, Evgeny Burnaev

LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization (2025.emnlp-main)

Challenge:	Recent studies show that a small subset of weights significantly impacts performance.
Approach:	They propose a Gaussian noise-injected fine-tuning method that updates only salient weights while injecting Gausssian into non-salient weight.
Outcome:	The proposed method outperforms full fine-tuning and PEFT methods under the same computational budget.

Copied to clipboard

Xujia Wang, Yunjia Qi, Bin Xu

PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning (2024.findings-acl)

Challenge:	Existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance.
Approach:	They propose a method that localizes and optimizes critical parameters during training . they propose 'LoSiA-Pro' which reduces training latency by 27% .
Outcome:	The proposed method achieves minimal performance drop compared to full fine-tuning while requiring the least training time across domain specialization and common-sense reasoning tasks.

Copied to clipboard

Zhisheng Lin, Han Fu, Chenghao Liu, Zhuo Li, Jianling Sun

Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning (2023.emnlp-main)

Challenge:	Parameter-efficient fine-tuning (PEFT) is an effective method for adapting pre-trained language models to various tasks efficiently.
Approach:	They propose a parameter-efficient fine-tuning framework that captures transferable knowledge as a weighted combination of adapters trained on source tasks.
Outcome:	The proposed method yields stable improvements over full fine-tuning and knowledge transferring methods on a broad range of tasks over 17 datasets.

Copied to clipboard

Sarkar Snigdha Sarathi Das, Haoran Zhang, Peng Shi, Wenpeng Yin, Rui Zhang

Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022.emnlp-main)

Challenge:	Named Entity Recognition, Relation Extraction, Semantic Role Labeling are examples of sequence labeling problems that require finetuning to the target format.
Approach:	They propose a dynamic sparse finetuning strategy that selectively focuses on a fraction of parameters, informed by feedback from highly regressing examples.
Outcome:	The proposed approach improves performance in low-resource settings and in extreme low-level settings.

Copied to clipboard

Hongyu Zhao, Hao Tan, Hongyuan Mei

ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts (2022.emnlp-main)

Challenge:	Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks . Previously proposed adapters are all feed-forward neural networks .
Approach:	They propose to use tiny-attention attention with extremely small per-head dimensionality as adapters to modify hidden states at each position . they propose to average multiple attention heads' weights during deployment to reduce its inference computation cost.
Outcome:	The proposed adapter outperforms other adapter-tuning methods on the GLUE benchmark . it uses attention with extremely small per-head dimensionality to modify hidden states .

Copied to clipboard

Akari Asai, Mohammadreza Salehi, Matthew Peters, Hannaneh Hajishirzi

KS-Lottery: Finding Certified Lottery Tickets for Multilingual Transfer in Large Language Models (2025.naacl-long)

Challenge:	a new multi-task, parameter-efficient language model tuning method learns to transfer knowledge across different tasks via a mixture of soft prompts.
Approach:	They propose a multi-task, parameter-efficient language model tuning method that uses soft prompts to learn to transfer knowledge across different tasks.
Outcome:	The proposed method outperforms prompt tuning and outperfies or matches fully fine-tuned tuning approaches that use 10 times more parameters.

Copied to clipboard

Fei Yuan, Chang Ma, Shuai Yuan, Qiushi Sun, Lei Li

Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization (2023.findings-emnlp)

Challenge:	Existing studies have shown that a small subset of parameters is highly effective in fine-tuning . prior work shows that there are a few additional parameters corresponding to an intrinsic dimension in a well-trained Large Language Model.
Approach:	They propose a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning.
Outcome:	The proposed method can find the certified winning tickets in the embedding layer, and fine-tuning on the found parameters is guaranteed to perform as well as full fine- tuning.

Copied to clipboard

Weixiao Zhou, Gengyao Li, Xianfu Cheng, Xinnian Liang, Junnan Zhu, Feifei Zhai, Zhoujun Li

Propulsion: Steering LLM with Tiny Fine-Tuning (2025.coling-main)

Challenge:	Existing methods for dialogue summarization only apply to specific scenarios and domains.
Approach:	They propose a pre-trained model specifically designed for multi-scenario multi-domain dialogue summarization.
Outcome:	The proposed model significantly outperforms state-of-the-art models on three dialogue summarization datasets from different scenarios and domains.

Copied to clipboard

Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat

LoRMA: Low-Rank Multiplicative Adaptation for LLMs (2025.findings-acl)

Challenge:	Propulsion is a parameter-efficient fine-tuning method that selectively re-scales specific dimensions of a pre-trained model without modifying the model’s parameters.
Approach:	They propose a parameter-efficient fine-tuning method that selectively re-scales specific dimensions of a pre-trained model without modifying the parameters.
Outcome:	The proposed method reduces parameter count from 355.3 million to 0.086 million while maintaining competitive performance across benchmarks.

Copied to clipboard

Harsh Bihany, Shubham Patel, Ashutosh Modi

When does Parameter-Efficient Transfer Learning Work for Machine Translation? (2022.emnlp-main)

Challenge:	Large Language Models have shown impressive generalization capabilities, but can be expensive to fine-tune due to high computational costs.
Approach:	They propose a low-rank multiplicative Adaptation technique that shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations.
Outcome:	The proposed approach overcomes computational complexity and rank bottlenecks in terms of matrix multiplication metrics.

Copied to clipboard

Ahmet Üstün, Asa Cooper Stickland

Fair and Argumentative Language Modeling for Computational Argumentation (2022.acl-long)

Challenge:	Prior work indicates that parametric fine-tuning methods may not work as well for machine translation (MT).
Approach:	They propose to use parameter-efficient fine-tuning methods to adapt large pre-trained models while only tuning a small number of parameters.
Outcome:	The proposed methods outperform full fine-tuning for many downstream tasks when the parameter budget corresponds to 10% of the model parameters.

Copied to clipboard

Carolin Holtermann, Anne Lauscher, Simone Ponzetto

HFT: Half Fine-Tuning for Large Language Models (2025.acl-long)

Challenge:	Recent work on stereotypical biases in semantic spaces is still in its infancy . we present a novel resource for bias measurement specifically tailored to argumentation .
Approach:	They propose a resource for bias measurement specifically tailored to argumentation . they use argumentative fine-tuning and debiasing to assess intrinsic bias .
Outcome:	The proposed approach is more sustainable and parameter-efficient than full fine-tuning . it can remove bias in general and argumentative language models while improving model performance in downstream tasks.

Copied to clipboard

Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, Hua Wu

Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation (2024.findings-emnlp)

Challenge:	Large language models (LLMs) with one or more fine-tuning phases can unlock various capabilities, but can be catastrophic forgetting during sequential training.
Approach:	They propose a method to regularly reset partial parameters to mitigate forgetting issues by using half fine-tuning instead of full fine-uning.
Outcome:	The proposed approach reduces the risk of catastrophic forgetting during training and the parametric knowledge lost during training may be overwhelmed by incoming training data.

Copied to clipboard

Branislav Pecher, Jan Cegin, Robert Belanec, Jakub Simko, Ivan Srba, Maria Bielikova

LoRaDA: Low-Rank Direct Attention Adaptation for Efficient LLM Fine-tuning (2025.findings-emnlp)

Challenge:	Despite the success of fine-tuning, it still displays model performance instability, especially with limited data.
Approach:	They propose a new mitigation strategy that leverages the strengths of ensembling, noise regularisation and model interpolation while retaining computational efficiency.
Outcome:	The proposed mitigation strategy outperforms the best performing mitigation strategy (Ensemble) while using only a fraction of its cost.

Copied to clipboard

Zhangming Li, Qinghao Hu, Yiqun Chen, Peisong Wang, Yifan Zhang, Jian Cheng

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models (2024.emnlp-main)

Challenge:	Recent advances in parameter-efficient fine-tuning techniques allow for adjustments to only a minor fraction of the parameters of language models.
Approach:	They propose a low-rank direct attention adapted method for efficient LLM fine-tuning . they propose LMAM, which can bring negative attention to self-attention modules .
Outcome:	The proposed method outperforms the full fine-tuning method by 2.1% on GLUE benchmark.

Copied to clipboard

Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Gauri Joshi

Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings (2025.emnlp-main)

Challenge:	federated fine-tuning of ODFMs is limited due to their limited size and system heterogeneity . emerging foundation models (FMs) have remarkable zero/few shot learning capabilities .
Approach:	They propose a federated fine-tuning method that leverages system and data heterogeneity at the edge.
Outcome:	a proposed method for federated fine-tuning improves performance on ODFMs . it allows heterogeneous LoRA ranks across clients for their individual system resources .

Copied to clipboard

Shiyu Li, Yang Tang, Ruijie Liu, Shi-Zhe Chen, Xi Chen

DiSec: Mitigating Backdoors in Pre-trained Language Models via Disentanglement of Adversarial Weights for Secure Fine-Tuning (2026.findings-acl)

Challenge:	Existing studies use LoRA to fine-tune existing LLMs, but this is limited by the data and training gap between them and embedding models.
Approach:	They propose a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder that integrates embeddings across different languages.
Outcome:	The proposed model improves performance on the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025).

Copied to clipboard

Sunanda Das, Qinghua Li

Robustness in NLP

Challenge:	Existing defenses rely on privileged assumptions, limiting their applicability in realistic settings.
Approach:	They propose a task-agnostic backdoor attack that contaminates pre-trained language models . authors propose auxiliary text purification framework that uses only clean auxiliary data .
Outcome:	The proposed framework suppresses attack success while preserving clean-task utility.

LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples (2026.findings-acl)

Copied to clipboard

Yezi Liu, Hanning Chen, Wenjun Huang, Yang Ni, Mohsen Imani

Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates (2026.acl-long)

Challenge:	Large Language Models encode vast factual knowledge, yet their inability to selectively forget specific information hinders privacy protection, bias mitigation, and post-deployment correction.
Approach:	They propose a LoRA-based negative-only unlearning framework that updates only low-rank adapters while freezing the backbone.
Outcome:	The proposed framework reduces computational cost by about an order of magnitude compared to full fine-tuning and memory-editing methods.

Copied to clipboard

Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, Nikolaos Aletras

An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)

Challenge:	Large language models underperform in languages absent or underrepresented in training data, creating barrier to equitable access for speakers worldwide.
Approach:	They propose a selective parameter update strategy that proactively preserves source knowledge by identifying critical parameters critical to maintaining source abilities.
Outcome:	Experiments in five typologically diverse languages show that SSU mitigates catastrophic forgetting.

Copied to clipboard

Zhongbin Xie, Thomas Lukasiewicz

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation (2025.emnlp-main)

Challenge:	Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems.
Approach:	They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation.
Outcome:	The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias.

Copied to clipboard

Junzhuo Li, Bo Wang, Xiuze Zhou, Xuming Hu

Outlier-weighed Layerwise Sampling for LLM Fine-tuning (2025.findings-acl)

Challenge:	Existing approaches to adapt Mixture-of-Experts models to multiple domains are prohibitive computation, cross-domain interference or require separate runs per domain.
Approach:	They propose a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models.
Outcome:	The proposed framework reduces forgetting by 89% compared to full fine-tuning as domains scale from 2 to 6 and achieves faster convergence than conventional methods.

Copied to clipboard

Pengxiang Li, Lu Yin, Xiaowei Gao, Shiwei Liu

Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain (2023.findings-emnlp)

Challenge:	Large Language Models (LLMs) are a powerful tool for processing complex natural language processing tasks.
Approach:	They propose an approach to fine-tune LLMs with outliers and a gradient low-rank projection to increase the number of fine-sampled layers without a proportional increase in memory costs.
Outcome:	The proposed approach outperforms baseline approaches while being more memory efficient.

Copied to clipboard

Isabella Olariu, Cedric Lothritz, Jacques Klein, Tegawendé Bissyandé, Siwen Guo, Shohreh Haddadan

NormAL LoRA: What is the perfect size? (2025.findings-emnlp)

Challenge:	Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular.
Approach:	They compare performance of financial BERT-like models to their fully fine-tuned counterparts by using parameter-efficient tuning methods.
Outcome:	The proposed approaches match full fine-tuning performance on common NLP tasks, but are less studied in finance.

Copied to clipboard

null Aastik, Topu Sai Meghana, Chinmay Prakash Kulkarni, Pragya Paramita Sahu

zFLoRA: Zero-Latency Fused Low-Rank Adapters (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) are crucial for enabling intelligent experiences across applications.
Approach:	They propose a low-rank adaptive localization method that uses rank-norm regularization to determine the optimal rank for each weight matrix.
Outcome:	NormAL LoRA reduces adapter parameters by 37% while preserving full fine-tuning performance.

Copied to clipboard

Dhananjaya Gowda, Seoha Song, Harshith Goka, Junhyun Lee

Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs (2025.emnlp-main)

Challenge:	Large language models (LLMs) are increasingly deployed with task-specific adapters catering to multiple downstream applications.
Approach:	They propose a low-latency fused low-rank adapter that introduces zero latency overhead on top of the base model.
Outcome:	The proposed adapter reduces the inference time of the model by 2.5x . the proposed adapters are tested on 18 different tasks on different platforms .

Copied to clipboard

Jun Bai, Minghao Tong, Yang Liu, Zixia Jia, Zilong Zheng

Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations (2025.emnlp-main)

Challenge:	Context faithfulness is essential for reliable reasoning in context-dependent scenarios.
Approach:	They propose a method that identifies and fine-tunes context-faithful experts . they propose 'context-faither fine- tuning' which selectively fine- tunes them .
Outcome:	The proposed method identifies experts with specialization in context utilization and improves context grounding.

Copied to clipboard

Diogo Tavares, David Semedo, Alexander Rudnicky, Joao Magalhaes

Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) neglect the embedding layer.
Approach:	They propose a PEFT approach that modifies input embeddings without altering hidden layers.
Outcome:	Experiments show that TinyTE modifies embeddings without altering hidden layers . the proposed approach achieves competitive performance while requiring 0.0001% of parameters .

Copied to clipboard

Antoine Simoulin, Namyong Park, Xiaoyi Liu, Grey Yang

Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models (2026.findings-acl)

Challenge:	Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass.
Approach:	They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens.
Outcome:	The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost .

Copied to clipboard

Kainan Liu, Yong Zhang, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao